Methods and systems for processing multi-media editing projects

ABSTRACT

Methods and systems of processing multi-media editing projects are described. In one embodiment, a software-implemented matrix switch is programmable to route multiple switch inputs to multiple switch outputs and to produce a single output that provides a data stream defined by a multi-media editing project. A request for one or more multi-media files for use in the multi-media editing project is generated by a user computer comprising part of a network where multi-media files are maintained in a network-accessible location. The request is intercepted and a determination is made as to whether a requested multi-media file is located on the user computer. The multi-media file is retrieved if the file is located on the user computer. If the file is not on the user computer, the requested file is sought from the network-accessible location. A multi-media editing project is then produced using the software-implemented matrix switch.

RELATED APPLICATION

This is a continuation of and claims priority to U.S. patent applicationSer. No. 09/732,086, filed on Dec. 6, 2000, the disclosure of which isincorporated by reference herein.

TECHNICAL FIELD

This invention generally relates to processing media content and, moreparticularly, to a system and related interfaces facilitating theprocessing of media content.

BACKGROUND

Recent advances in computing power and related technology have fosteredthe development of a new generation of powerful software applications.Gaming applications, communications applications, and multimediaapplications have particularly benefited from increased processing powerand clocking speeds. Indeed, once the province of dedicated, specialtyworkstations, many personal computing systems now have the capacity toreceive, process and render multimedia objects (e.g., audio and videocontent). While the ability to display (receive, process and render)multimedia content has been around for a while, the ability for astandard computing system to support true multimedia editingapplications is relatively new.

In an effort to satisfy this need, Microsoft Corporation introduced aninnovative development system supporting advanced user-definedmultimedia editing functions. An example of this architecture ispresented in U.S. Pat. No. 5,913,038 issued to Griffiths and commonlyowned by the assignee of the present invention, the disclosure of whichis expressly incorporated herein by reference.

In the '038 patent, Griffiths introduced the an application programinterface which, when exposed to higher-level development applications,enables a user to graphically construct a multimedia processing projectby piecing together a collection of “filters” exposed by the interface.The interface described therein is referred to as a filter graphmanager. The filter graph manager controls the data structure of thefilter graph and the way data moves through the filter graph. The filtergraph manager provides a set of component object model (COM) interfacesfor communication between a filter graph and its application. Filters ofa filter graph architecture are preferably implemented as COM objects,each implementing one or more interfaces, each of which contains apredefined set of functions, called methods. Methods are called by anapplication program or other component objects in order to communicatewith the object exposing the interface. The application program can alsocall methods or interfaces exposed by the filter graph manager object.

Filter graphs work with data representing a variety of media (ornon-media) data types, each type characterized by a data stream that isprocessed by the filter components comprising the filter graph. A filterpositioned closer to the source of the data is referred to as anupstream filter, while those further down the processing chain isreferred to as a downstream filter. For each data stream that the filterhandles it exposes at least one virtual pin (i.e., distinguished from aphysical pin such as one might find on an integrated circuit). A virtualpin can be implemented as a COM object that represents a point ofconnection for a unidirectional data stream on a filter. Input pinsrepresent inputs and accept data into the filter, while output pinsrepresent outputs and provide data to other filters. Each of the filtersinclude at least one memory buffer, wherein communication of the mediastream between filters is often accomplished by a series of “copy”operations from one filter to another.

As introduced in Griffiths, a filter graph has three different types offilters: source filters, transform filters, and rendering filters. Asource filter is used to load data from some source; a transform filterprocesses and passes data; and a rendering filter renders data to ahardware device or other locations (e.g., saved to a file, etc.). Anexample of a filter graph for a simplistic media rendering process ispresented with reference to FIG. 1.

FIG. 1 graphically illustrates an example filter graph for renderingmedia content. As shown, the filter graph 100 is comprised of aplurality of filters 102-114, which read, process (transform) and rendermedia content from a selected source file. As shown, the filter graphincludes each of the types of filters described above, interconnected ina linear fashion.

Products utilizing the filter graph have been well received in themarket as it has opened the door to multimedia editing using otherwisestandard computing systems. It is to be appreciated, however, that theconstruction and implementation of the filter graphs are computationallyintensive and expensive in terms of memory usage. Even the most simpleof filter graphs requires and abundance of memory to facilitate the copyoperations required to move data between filters. Complex filter graphscan become unwieldy, due in part to the linear nature of prior artfilter graph architecture. Moreover, it is to be appreciated that thefilter graphs themselves consume memory resources, thereby compoundingthe issue introduced above.

Thus, what is required is a filter graph architecture which reduces thecomputational and memory resources required to support even the mostcomplex of multimedia projects. Indeed, what is required is adynamically reconfigurable multimedia editing system and relatedmethods, unencumbered by the limitations described above. Just such asystem and methods are disclosed below.

SUMMARY

Methods and systems of processing multi-media editing projects aredescribed. In one embodiment, a software-implemented matrix switch isprogrammable to route multiple switch inputs to multiple switch outputsand to produce a single output that provides a data stream defined by amulti-media editing project. A request for one or more multi-media filesfor use in the multi-media editing project is generated by a usercomputer comprising part of a network where multi-media files aremaintained in a network-accessible location. The request is interceptedand a determination is made as to whether a requested multi-media fileis located on the user computer. The multi-media file is retrieved ifthe file is located on the user computer. If the file is not on the usercomputer, the requested file is sought from the network-accessiblelocation. A multi-media editing project is then produced using thesoftware-implemented matrix switch.

BRIEF DESCRIPTION OF THE DRAWINGS

The same reference numbers are used throughout the figures to referencelike components and features.

FIG. 1 is a graphical representation of a conventional filter graphrepresenting a user-defined development project.

FIG. 2 is a block diagram of a computing system incorporating theteachings of the described embodiment.

FIG. 3 is a block diagram of an example software architectureincorporating the teachings of the described embodiment.

FIG. 4 is a graphical illustration of an example software-enabled matrixswitch, according to an exemplary embodiment.

FIG. 5 is a graphical representation of a data structure comprising aprogramming grid to selectively couple one or more of a scalableplurality of input pins to a scalable plurality of output pins of thematrix switch filter, in accordance with one aspect of the describedembodiment.

FIG. 6 is a graphical illustration denoting shared buffer memory betweenfilters, according to one aspect of the described embodiment.

FIG. 7 is a flow chart of an example method for generating a filtergraph, in accordance with one aspect of the described embodiment.

FIG. 8 is a flow chart of an example method for negotiating bufferrequirements between at least two adjacent filters, according to oneaspect of the described embodiment.

FIG. 9 graphically illustrates an overview of a process that takes auser-defined editing project and composites a data structure that can beused to program the matrix switch.

FIG. 10 graphically illustrates the project of FIG. 9 in greater detail.

FIG. 11 shows an exemplary matrix switch dynamically generated insupport of the project developed in FIGS. 9 and 10, according to onedescribed embodiment.

FIG. 12 illustrates a graphic representation of an exemplary datastructure that represents the project of FIG. 10, according to onedescribed embodiment.

FIGS. 13-18 graphically illustrate various states of a matrix switchprogramming grid at select points in processing the project of FIGS. 9and 10 through the matrix switch, in accordance with one describedembodiment.

FIG. 19 is a flow chart of an example method for processing mediacontent, in accordance with one described embodiment.

FIG. 20 illustrates an example project with a transition and an effect,in accordance with one described embodiment.

FIG. 21 shows an exemplary data structure in the form of a hierarchicaltree that represents the project of FIG. 20.

FIGS. 22 and 23 graphically illustrate an example matrix switchprogramming grid associated with the project of FIG. 20 at select pointsin time, according to one described embodiment.

FIG. 24 shows an example matrix switch dynamically generated andconfigured as the grid of FIGS. 22 and 23 was being processed, inaccordance with one described embodiment.

FIG. 25 shows an exemplary project in accordance with one describedembodiment.

FIG. 26 graphically illustrates an example audio editing project,according to one described embodiment.

FIG. 27 depicts an example matrix switch programming grid associatedwith the project of FIG. 26.

FIG. 28 shows an example matrix switch dynamically generated andconfigured in accordance with the programming grid of FIG. 27 to performthe project of FIG. 26, according to one described embodiment.

FIG. 29 illustrates an exemplary media processing project incorporatinganother media processing project as a composite, according to yetanother described embodiment.

FIG. 30 graphically illustrates an example data structure in the form ofa hierarchical tree structure that represents the project of FIG. 29.

FIGS. 31-36 graphically illustrate various matrix switch programminggrid states at select points in generating and configuring the matrixswitch to implement the media processing of FIG. 29.

FIG. 38 illustrates an example matrix switch suitable for use in themedia processing project of FIG. 29, according to one describedembodiment.

FIG. 38 a graphically illustrates an example data structure in the formof a hierarchical tree structure that represents a project that isuseful in understanding composites in accordance with the describedembodiments.

FIG. 39 is a flow diagram that describes steps in a method in accordancewith one described embodiment.

FIG. 40 shows an exemplary system in which various embodiments can beimplemented.

FIG. 41 is a flow diagram that describes steps in a method in accordancewith one described embodiment.

FIG. 42 illustrates an exemplary multi-media file locator object inaccordance with one embodiment.

DETAILED DESCRIPTION

Related Applications

This application is related to the following commonly-filed U.S. PatentApplications, all of which are commonly assigned to Microsoft Corp., thedisclosures of which are incorporated by reference herein: applicationSer. No. 09/731,560, entitled “An Interface and Related Methods forReducing Source Accesses in a Development System”, naming Daniel J.Miller and Eric H. Rudolph as inventors; application Ser. No.09/732,084, entitled “A System and Related Interfaces Supporting theProcessing of Media Content”, naming Daniel J. Miller and Eric H.Rudolph as inventors; application Ser. No. 09/731,490, entitled “ASystem and Related Methods for Reducing Source Filter Invocation in aDevelopment Project”, naming Daniel J. Miller and Eric H. Rudolph asinventors; application Ser. No. 09/732,452, entitled “A System andRelated Methods for Reducing Memory Requirements of a Media ProcessingSystem”, naming Daniel J. Miller and Eric H. Rudolph as inventors;application Ser. No. 09/731,529, entitled “A System and Related Methodsfor Reducing the Instances of Source Files in a Filter Graph”, namingDaniel J. Miller and Eric H. Rudolph as inventors; application Ser. No.09/732,087, entitled “An Interface and Related Methods for DynamicallyGenerating a Filter Graph in a Development System”, naming Daniel J.Miller and Eric H. Rudolph as inventors; application Ser. No.09/732,090, entitled “A System and Related Methods for Processing AudioContent in a Filter Graph”, naming Daniel J. Miller and Eric H. Rudolphas inventors; application Ser. No. 09/732,085, entitled “A System andMethods for Generating an Managing Filter Strings in a Filter Graph”,naming Daniel J. Miller and Eric H. Rudolph as inventors; applicationSer. No. 09/731,491, entitled “Methods and Systems for Processing MediaContent”, naming Daniel J. Miller and Eric H. Rudolph as inventors;application Ser. No. 09/731,563, entitled “Systems for Managing MultipleInputs and Methods and Systems for Processing Media Content”, namingDaniel J. Miller and Eric H. Rudolph as inventors; application Ser. No.09/731,892, entitled “Methods and Systems for Implementing DynamicProperties on Objects that Support Only Static Properties”, namingDaniel J. Miller and David Maymudes as inventors; application Ser. No.09/732,089, entitled “Methods and Systems for Efficiently ProcessingCompressed and Uncompressed Media Content”, naming Daniel J. Miller andEric H. Rudolph as inventors; application Ser. No. 09/731,581, entitled“Methods and Systems for Effecting Video Transitions Represented ByBitmaps”, naming Daniel J. Miller and David Maymudes as inventors; andapplication Ser. No. 09/732,372, entitled “Methods and Systems forMixing Digital Audio Signals”, naming Eric H. Rudolph as inventor.

Various described embodiments concern an application program interfaceassociated with a development system. According to one exampleimplementation, the interface is exposed to a media processingapplication to enable a user to dynamically generate complex mediaprocessing tasks, e.g., editing projects. In the discussion herein,aspects of the invention are developed within the general context ofcomputer-executable instructions, such as program modules, beingexecuted by one or more conventional computers. Generally, programmodules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. Moreover, those skilled in the art will appreciatethat the invention may be practiced with other computer systemconfigurations, including hand-held devices, personal digitalassistants, multiprocessor systems, microprocessor-based or programmableconsumer electronics, network PCs, minicomputers, mainframe computers,and the like. In a distributed computer environment, program modules maybe located in both local and remote memory storage devices. It is noted,however, that modification to the architecture and methods describedherein may well be made without deviating from spirit and scope of thepresent invention. Moreover, although developed within the context of amedia processing system paradigm, those skilled in the art willappreciate, from the discussion to follow, that the application programinterface may well be applied to other development systemimplementations. Thus, the media processing system described below isbut one illustrative implementation of a broader inventive concept.

Example System Architecture

FIG. 2 illustrates an example of a suitable computing environment 200 onwhich the system and related methods for processing media content may beimplemented.

It is to be appreciated that computing environment 200 is only oneexample of a suitable computing environment and is not intended tosuggest any limitation as to the scope of use or functionality of themedia processing system. Neither should the computing environment 200 beinterpreted as having any dependency or requirement relating to any oneor combination of components illustrated in the exemplary computingenvironment 200.

The media processing system is operational with numerous other generalpurpose or special purpose computing system environments orconfigurations. Examples of well known computing systems, environments,and/or configurations that may be suitable for use with the mediaprocessing system include, but are not limited to, personal computers,server computers, thin clients, thick clients, hand-held or laptopdevices, multiprocessor systems, microprocessor-based systems, set topboxes, programmable consumer electronics, network PCs, minicomputers,mainframe computers, distributed computing environments that include anyof the above systems or devices, and the like.

In certain implementations, the system and related methods forprocessing media content may well be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, etc. that performparticular tasks or implement particular abstract data types. The mediaprocessing system may also be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including memory storage devices.

In accordance with the illustrated example embodiment of FIG. 2computing system 200 is shown comprising one or more processors orprocessing units 202, a system memory 204, and a bus 206 that couplesvarious system components including the system memory 204 to theprocessor 202.

Bus 206 is intended to represent one or more of any of several types ofbus structures, including a memory bus or memory controller, aperipheral bus, an accelerated graphics port, and a processor or localbus using any of a variety of bus architectures. By way of example, andnot limitation, such architectures include Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnects (PCI) buss also known asMezzanine bus.

Computer 200 typically includes a variety of computer readable media.Such media may be any available media that is locally and/or remotelyaccessible by computer 200, and it includes both volatile andnon-volatile media, removable and non-removable media.

In FIG. 2, the system memory 204 includes computer readable media in theform of volatile, such as random access memory (RAM) 210, and/ornon-volatile memory, such as read only memory (ROM) 208. A basicinput/output system (BIOS) 212, containing the basic routines that helpto transfer information between elements within computer 200, such asduring start-up, is stored in ROM 208. RAM 210 typically contains dataand/or program modules that are immediately accessible to and/orpresently be operated on by processing unit(s) 202.

Computer 200 may further include other removable/non-removable,volatile/non-volatile computer storage media. By way of example only,FIG. 2 illustrates a hard disk drive 228 for reading from and writing toa non-removable, non-volatile magnetic media (not shown and typicallycalled a “hard drive”), a magnetic disk drive 230 for reading from andwriting to a removable, non-volatile magnetic disk 232 (e.g., a “floppydisk”), and an optical disk drive 234 for reading from or writing to aremovable, non-volatile optical disk 236 such as a CD-ROM, DVD-ROM orother optical media. The hard disk drive 228, magnetic disk drive 230,and optical disk drive 234 are each connected to bus 206 by one or moreinterfaces 226.

The drives and their associated computer-readable media providenonvolatile storage of computer readable instructions, data structures,program modules, and other data for computer 200. Although the exemplaryenvironment described herein employs a hard disk 228, a removablemagnetic disk 232 and a removable optical disk 236, it should beappreciated by those skilled in the art that other types of computerreadable media which can store data that is accessible by a computer,such as magnetic cassettes, flash memory cards, digital video disks,random access memories (RAMs), read only memories (ROM), and the like,may also be used in the exemplary operating environment.

A number of program modules may be stored on the hard disk 228, magneticdisk 232, optical disk 236, ROM 208, or RAM 210, including, by way ofexample, and not limitation, an operating system 214, one or moreapplication programs 216 (e.g., multimedia application program 224),other program modules 218, and program data 220. In accordance with theillustrated example embodiment of FIG. 2, operating system 214 includesan application program interface embodied as a render engine 222. Aswill be developed more fully below, render engine 222 is exposed tohigher-level applications (e.g., 216) to automatically assemble filtergraphs in support of user-defined development projects, e.g., mediaprocessing projects. Unlike conventional media processing systems,however, render engine 222 utilizes a scalable, dynamicallyreconfigurable matrix switch to reduce filter graph complexity, therebyreducing the computational and memory resources required to complete adevelopment project. Various aspects of the innovative media processingsystem represented by a computer 200 implementing the innovative renderengine 222 will be developed further, below.

Continuing with FIG. 2, a user may enter commands and information intocomputer 200 through input devices such as keyboard 238 and pointingdevice 240 (such as a “mouse”). Other input devices may include aaudio/video input device(s) 253, a microphone, joystick, game pad,satellite dish, serial port, scanner, or the like (not shown). These andother input devices are connected to the processing unit(s) 202 throughinput interface(s) 242 that is coupled to bus 206, but may be connectedby other interface and bus structures, such as a parallel port, gameport, or a universal serial bus (USB).

A monitor 256 or other type of display device is also connected to bus206 via an interface, such as a video adapter 244. In addition to themonitor, personal computers typically include other peripheral outputdevices (not shown), such as speakers and printers, which may beconnected through output peripheral interface 246.

Computer 200 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer250. Remote computer 250 may include many or all of the elements andfeatures described herein relative to computer 200 including, forexample, render engine 222 and one or more development applications 216utilizing the resources of render engine 222.

As shown in FIG. 2. computing system 200 is communicatively coupled toremote devices (e.g., remote computer 250) through a local area network(LAN) 251 and a general wide area network (WAN) 252. Such networkingenvironments are commonplace in offices, enterprise-wide computernetworks, intranets, and the Internet.

When used in a LAN networking environment, the computer 200 is connectedto LAN 251 through a suitable network interface or adapter 248. Whenused in a WAN networking environment, the computer 200 typicallyincludes a modem 254 or other means for establishing communications overthe WAN 252. The modem 254, which may be internal or external, may beconnected to the system bus 206 via the user input interface 242, orother appropriate mechanism.

In a networked environment, program modules depicted relative to thepersonal computer 200, or portions thereof, may be stored in a remotememory storage device. By way of example, and not limitation, FIG. 2illustrates remote application programs 216 as residing on a memorydevice of remote computer 250. It will be appreciated that the networkconnections shown and described are exemplary and other means ofestablishing a communications link between the computers may be used.

Turning next to FIG. 3, a block diagram of an example development systemarchitecture is presented, in accordance with one embodiment of thepresent invention. In accordance with the illustrated example embodimentof FIG. 3, development system 300 is shown comprising one or moreapplication program(s) 216 coupled to render engine 222 via anappropriate communications interface 302. As used herein, applicationprogram(s) 216 are intended to represent any of a wide variety ofapplications which may benefit from use of render engine 222 such as,for example a media processing application 224.

The communications interface 302 is intended to represent any of anumber of alternate interfaces used by operating systems to exposeapplication program interface(s) to applications. According to oneexample implementation, interface 302 is a component object model (COM)interface, as used by operating systems offered by MicrosoftCorporation. As introduced above, COM interface 302 provides a means bywhich the features of the render engine 222, to be described more fullybelow, are exposed to an application program 216.

In accordance with the illustrated example implementation of FIG. 3,render engine 222 is presented comprising source filter(s) 304A-N,transform filter(s) 306A-N and render filter 310, coupled togetherutilizing virtual pins to facilitate a user-defined media processingproject. According to one implementation, the filters of system 300 aresimilar to the filters exposed in conventional media processing systems.According to one implementation, however, filters are not coupled viasuch interface pins. Rather, alternate implementations are envisionedwherein individual filters (implemented as objects) make calls to otherobjects, under the control of the render engine 222, for the desiredinput. Unlike conventional systems, however, render engine 222 exposes ascalable, dynamically reconfigurable matrix switch filter 308,automatically generated and dynamically configured by render engine 222to reduce the computational and memory resource requirements oftenassociated with development projects. As introduced above, the pins(input and/or output) are application interface(s) designed tocommunicatively couple other objects (e.g., filters).

In accordance with the example implementation of a media processingsystem, an application communicates with an instance of render engine222 when the application 216 wants to process streaming media content.Render engine 222 selectively invokes and controls an instance of filtergraph manager (not shown) to automatically create a filter graph byinvoking the appropriate filters (e.g., source, transform andrendering). As introduced above, the communication of media contentbetween filters is achieved by either (1) coupling virtual output pinsof one filter to the virtual input pins of requesting filter; or (2) byscheduling object calls between appropriate filters to communicate therequested information. As shown, source filter 304 receives streamingdata from the invoking application or an external source (not shown). Itis to be appreciated that the streaming data can be obtained from a fileon a disk, a network, a satellite feed, an Internet server, a videocassette recorder, or other source of media content. As introducedabove, transform filter(s) 306 take the media content and processes itin some manner, before passing it along to render filter 310. As usedherein, transform filter(s) 306 are intended to represent a wide varietyof processing methods or applications that can be performed on mediacontent. In this regard, transform filter(s) 306 may well include asplitter, a decoder, a sizing filter, a transition filter, an effectsfilter, and the like. The function of each of these filters is describedmore fully in the Griffiths application, introduced above, and generallyincorporated herein by reference. The transition filter, as used herein,is utilized by render engine 222 to transition the rendered output froma first source to a second source. The effect filter is selectivelyinvoked to introduce a particular effect (e.g., fade, wipe, audiodistortion, etc.) to a media stream.

In accordance with one aspect of the embodiment, to be described morefully below, matrix switch filter 308 selectively passes media contentfrom one or more of a scalable plurality of input(s) to a scalableplurality of output(s). Moreover, matrix switch 308 also supportsimplementation of a cascaded architecture utilizing feedback paths,i.e., wherein transform filters 306B, 306C, etc. coupled to the outputof matrix switch 308 are dynamically coupled to one or more of thescalable plurality of matrix switch input(s). An example of thiscascaded filter graph architecture is introduced in FIG. 3, and furtherexplained in example implementations, below.

Typically, media processed through source, transform and matrix switchfilters are ultimately passed to render filter 310, which provides thenecessary interface to a hardware device, or other location that acceptsthe renderer output format, such as a memory or disk file, or arendering device.

FIG. 4 is a graphical illustration of an example software-enabled matrixswitch 308, according to one example embodiment of the presentinvention. As shown, the matrix switch 308 is comprised of a scalableplurality of input(s) 402 and a scalable plurality of output(s) 404,wherein any one or more of the input(s) 402 may be iteratively coupledto any one or more of the output(s) 404, based on the content of thematrix switch programming grid 406, automatically generated by renderengine 222. According to an alternate implementation introduced above,switch matrix 308 is programmed by render engine 222 to dynamicallygenerate object calls to communicate media content between filters. Inaddition, according to one implementation, matrix switch 308 includes aplurality of input/output (I/O) buffers 408, as well as means formaintaining source, or media time 410 and/or timeline, or project time412. It is to be appreciated, however, that in alternate implementationsmatrix switch 308 does not maintain both source and project times,relying on an upstream filter to convert between these times. As will bedeveloped more fully below, matrix switch 308 dynamically couples one ormore of the scalable plurality of inputs 402 to one or more of thescalable plurality of outputs 404 based, at least in part, on the mediatime 410 and/or the project time 412 and the content of matrix switchprogramming grid 406. In this regard, matrix switch 308 may becharacterized as time-aware, supporting such advanced editing featuresas searching/seeking to a particular point (e.g., media time) in themedia content, facilitating an innovative buffering process utilizingI/O buffers 408 to facilitate look-ahead processing of media content,and the like. Thus, it will be appreciated given the discussion tofollow that introduction of the matrix switch 308 provides a user withan editing flexibility that was heretofore unavailable in a personalcomputer-based media processing system.

As introduced above, the inputs 402 and outputs 404 of matrix switch 308are interfaces which facilitate the time-sensitive routing of data(e.g., media content) in accordance with a user-defined developmentproject. Matrix switch 308 has a scalable plurality of inputs 402 andoutputs 404, meaning that the number of inputs 402 and outputs 404 areindividually generated to satisfy a given editing project. Insofar aseach of the inputs/outputs (I/O) has an associated transfer buffer(preferably shared with an adjacent filter) to communicate mediacontent, the scalability of the input/output serves to reduce theoverall buffer memory consumed by an editing project. According to oneimplementation, output 1 is generally reserved as a primary output,e.g., coupled to a rendering filter (not shown).

According to one implementation, for each input 402 and output 404,matrix switch 308 attempts to be the allocator, or manager of the bufferassociated with the I/O(s) shared with adjacent filters. One reason isto ensure that all of the buffers are of the same size and share commonattributes so that a buffer associated with any input 402 may be sharedwith any output 404, thereby reducing the need to copy memory contentsbetween individual buffers associated with such inputs/outputs. Ifmatrix switch 308 cannot be an allocator for a given output (404),communication from an input (402) to that output is performed using aconventional memory copy operation between the individual buffersassociated with the select input/output.

As introduced above, the matrix switch programming grid 406 isdynamically generated by render engine 222 based, at least in part, onthe user-defined development project. As will be developed below, renderengine 222 invokes an instance of filter graph manager to assembles atree structure of an editing project, noting dependencies betweensource, filters and time to dynamically generate the programming grid406. A data structure comprising an example programming grid 406 isintroduced with reference to FIG. 5, below.

Turning briefly to FIG. 5, a graphical representation of a datastructure comprising an example programming grid 406 is presented, inaccordance with one embodiment of the present invention. In accordancewith the illustrated example embodiment of FIG. 5, programming grid 406is depicted as a two-dimensional data structure comprising a columnalong the y-axis 502 of the grid denoting input pins associated with acontent chain (e.g., series of filters to process media content) of thedevelopment project. The top row along the x-axis 504 of the datastructure denotes project time. With these grid “borders”, the body 506of the grid 406 is populated with output pin assignments, denoting whichinput pin is coupled to which output pin during execution of thedevelopment project. In this way, render engine 222 dynamicallygenerates and facilitates matrix switch 308. Those skilled in the artwill appreciate, however, that data structures of greater or lessercomplexity may well be used in support of the programming grid 406without deviating from the spirit and scope of the present invention.

Returning to FIG. 4, matrix switch 308 is also depicted with a pluralityof input/output buffers 408, shared among all of the input(s)/ouptut(s)(402, 404) to facilitate advanced processing features. That is, whilenot required to implement the core features of matrix switch 308, I/Obuffers 408 facilitate a number of innovative performance enhancingfeatures to improve the performance (or at least the user's perceptionof performance) of the processing system, thereby providing an improveduser experience. According to one implementation, I/O buffers 408 areseparate from the buffers assigned to each individual input and outputpin in support of communication through the switch. According to oneimplementation, I/O buffers 408 are primarily used to foster look-aheadprocessing of the project. Assume, for example, that a large portion ofthe media processing project required only 50% of the availableprocessing power, while some smaller portion required 150% of theavailable processing power. Implementation of the shared I/O buffers 408enable filter graph manager to execute tasks ahead of schedule andbuffer this content in the shared I/O buffers 408 until required. Thus,when execution of the filter graph reaches a point where more than 100%of the available processing power is required, the processing system cancontinue to supply content from the I/O buffers 408, while the systemcompletes execution of the CPU-intensive tasks. If enough shared bufferspace is provided, the user should never know that some tasks were notperformed in real-time. According to one implementation, shared buffers408 are dynamically split into two groups by render engine 222, a firstgroup supports the input(s) 402, while a second (often smaller) group isused in support of a primary output (e.g., output pin 1) to facilitate asecond, independent output processing thread. The use of an independentoutput buffers the render engine from processing delays that might occurin upstream and/or downstream filters, as discussed above. It will beappreciated by those skilled in the art that such that matrix switch 308and the foregoing described architecture beneficially suited to supportmedia streaming applications.

As introduced above, the filter graph is time-aware in the sense thatmedia (source) time and project execution time are maintained. Accordingto one implementation, matrix switch 308 maintains at least the projectclock, while an upstream filter maintains the source time, convertingbetween source and project time for all downstream filters (i.e.,including the matrix switch 308). According to one implementation, theframe rate converter filter of a filter graph is responsible forconverting source time to project time, and vice versa, i.e., supportingrandom seeks, etc. Alternatively, matrix switch 308 utilizes anintegrated set of clock(s) to independently maintain project and mediatimes.

Having introduced the architectural and operational elements of matrixswitch filter 308, FIG. 6 graphically illustrates an example filtergraph implementation incorporating the innovative matrix switch 308. Inaccordance with the illustrated example embodiment, filter graph 600 isgenerated by render engine 222 in response to a user defined developmentproject. Unlike the lengthy linear filter graphs typical of conventiondevelopment systems however, filter graph 600 is shown incorporating amatrix switch filter 308 to recursively route the pre-processed content(e.g., through filters 602, 606, 610, 614 and 618, described more fullybelow) through a user-defined number of transform filters including, forexample, transition filter(s) 620 and effects filter(s) 622. Moreover,as will be developed more fully below, the scalable nature of matrixswitch filter 308 facilitates such iterative processing for any numberof content threads, tracks or compositions.

According to one implementation, a matrix switch filter 308 can onlyprocess one type of media content, of the same size and at the sameframe-rate (video) or modulation type/schema (audio). Thus, FIG. 6 isdepicted comprising pre-processing filters with a parser filter 606 toseparate, independent content type(s) (e.g., audio content and videocontent), wherein one of the media types would be processed along adifferent path including a separate instance of matrix switch 308. Thus,in accordance with the illustrated example embodiment of a mediaprocessing system, processing multimedia content including audio andvideo would utilize two (2) matrix switch filters 308, one dedicated toaudio processing (not shown) and one dedicated to video processing. Thatis not to say, however, that multiple switch filters 308 could not beused (e.g., two each for audio and video) for each content type inalternate implementations. Similarly, it is anticipated that inalternate implementations a matrix switch 308 that accepts multiplemedia types could well be used without deviating from the spirit andscope of the present invention.

In addition filter graph 600 includes a decoder filter 610 to decode themedia content. Resize filter 614 is employed when matrix switch 308 isto receive content from multiple sources, ensuring that the size of thereceived content is the same, regardless of the source. According to oneimplementation, resize filter 614 is selectively employed in videoprocessing paths to adjust the media size of content from one or moresources to a user-defined level. Alternatively, resizer filter 614adjusts the media size to the largest size provided by any one or moremedia sources. That is, if, for example, render engine 222 identifiesthe largest required media size (e.g., 1270×1040 video pixels per frame)and, for any content source not providing content at this size, thecontent is modified (e.g., stretched, packed, etc.) to fill this sizerequirement. The frame rate converter (FRC) and pack filter 618,introduced above, ensures that video content from the multiple sourcesis arriving at the same frame rate, e.g., ten (10) frames per second. Asintroduced above, the FRC also maintains the distinction between sourcetime and project time.

In accordance with one aspect of the present invention, filter graph 600is depicted utilizing a single, negotiated buffer 604, 608, 612, 616,etc. between adjacent filters. In this regard, render engine 222 reducesthe buffer memory requirements in support of a development project.

From the point of pre-processing (filters 602, 606, 610, 614, 618),rather than continue a linear filter graph incorporating all of thetransition 620 and effect 622 filter(s), render engine 222 utilizes acascade architecture, recursively passing media content through thematrix switch 308 to apply to the transform filter(s) (e.g., 620, 622,etc.) to complete the execution of the development project. It will beappreciated by those skilled in the art that the ability to recursivelypass media content through one or more effect and/or transition filtersprovided by the matrix switch filter 308 greatly reduces the perceivedcomplexity of otherwise large filter graphs, while reducing memory andcomputational overhead.

Turning to FIG. 7, a flow chart of an example method for generating afilter graph is presented, in accordance with one aspect of the presentinvention. The method 700 begins with block 702 wherein render engine222 receives an indication to generate a filter graph representing auser-defined development project (e.g., a media editing project).According to one example implementation, the indication is received froman application 224 via COM interface(s) 302.

In block 704, render engine 222 facilitates generation of the editingproject, identifying the number and type of media sources selected bythe user. In block 706, based at least in part on the number and/or typeof media sources, filter graph manger 222 exposes source, transform andrendering filter(s) to effect a user defined media processing project,while beginning to establish a programming grid 406 for the matrixswitch filter 308.

In block 708, reflecting user editing instructions, render engine 222completes the programming grid 406 for matrix switch 308, identifyingwhich inputs 402 are to be coupled to which outputs 404 at particularproject times.

Based, at least in part, on the programming grid 406 render engine 222generates a matrix switch filter 308 with an appropriate number of input402 and output 404 pins to effect the project, and assembles the filtergraph, block 710.

In block 712, to reduce the buffer memory requirements for theprocessing project, the render engine 222 instructs the filterspopulating the filter graph to (re)negotiate buffer memory requirementsbetween filters. That is, adjacent filters attempt to negotiate a sizeand attribute standard so that a single buffer can be utilized to coupleeach an output pin of one filter to an input pin of a downstream filter.An example implementation of the buffer negotiation process of block 712is presented in greater detail with reference to FIG. 8.

Turning briefly to FIG. 8, an example method of negotiating bufferrequirements between adjacent filters is presented, in accordance withone example implementation of the present invention. Once the finalconnection is established to matrix switch 308, matrix switch 308identifies the maximum buffer requirements for any filter coupled to anyof its pins (input 402 and/or output 404), block 802. According to oneimplementation, the maximum buffer requirements are defined as thelowest common multiple of buffer alignment requirements, and the maximumof all the pre-fix requirements of the filter buffers.

In block 804, matrix switch 308 selectively removes one or more existingfilter connections to adjacent filters. Matrix switch 308 thenreconnects all of its pins to adjacent filters using a common buffersize between each of the pins, block 806. In block 808, matrix switch308 negotiates to be the allocator for all of its pins (402, 404). Ifthe matrix switch 308 cannot, for whatever reason, be the allocator forany of its input pins 402 minimal loss to performance is encountered, asthe buffer associated with the input pin will still be compatible withany downstream filter (i.e., coupled to an output pin) and, thus, thebuffer can still be passed to the downstream filter without requiring amemory copy operation. If, however, matrix switch 308 cannot be anallocator for one of its output pins 404, media content must then betransferred to at least the downstream filter associated with thatoutput pin using a memory copy operation, block 810.

In block 812, once the matrix switch 308 has re-established itsconnection to adjacent filters, render engine 222 restores theconnection in remaining filters using negotiated buffer requirementsemanating from the matrix switch filter 308 buffer negotiations. Oncethe connections throughout the filter graph have been reconnected, theprocess continues with block 714 of FIG. 7.

In block 714 (FIG. 7), have re-established the connections betweenfilters, render engine 222 is ready to implement a user's instruction toexecute the media processing project.

Example Operation and Implementation(s)

The matrix switch described above is quite useful in that it allowsmultiple inputs to be directed to multiple outputs at any one time.These input can compete for a matrix switch output. The embodimentsdescribed below permit these competing inputs to be organized so thatthe inputs smoothly flow through the matrix switch to provide a desiredoutput. And, while the inventive programming techniques are described inconnection with the matrix switch as such is employed in the context ofmulti-media editing projects, it should be clearly understood thatapplication of the inventive programming techniques and structuresshould not be so limited only to application in the field of multi-mediaediting projects or, for that matter, multi-media applications or datastreams. Accordingly, the principles about to be discussed can beapplied to other fields of endeavor in which multiple inputs can becharacterized as competing for a particular output during a common timeperiod.

In the multi-media example below, the primary output of the matrixswitch is a data stream that defines an editing project that has beencreated by a user. Recall that this editing project can include multipledifferent sources that are combined in any number of different ways, andthe sources that make up a project can comprise audio sources, videosources, or both. The organization of the inputs and outputs of thematrix switch are made manageable, in the examples described below, by adata structure that permits the matrix switch to be programmed.

FIG. 9 shows an overview of a process that takes a user-defined editingproject and renders from it a data structure that can be used to programthe matrix switch.

Specifically, a user-defined editing project is shown generally at 900.Typically, when a user creates an editing project, they can select froma number of different multimedia clips that they can then assemble intoa unique presentation. Each individual clip represents a source ofdigital data or a source stream (e.g., multimedia content). Projects caninclude one or more sources 902. In defining their project, a user canoperate on sources in different ways. For example, video sources canhave transitions 904 and effects 906 applied on them. A transitionobject is a way to change between two or more sources. As discussedabove, a transition essentially receives as input, two or more streams,operates on them in some way, and produces a single output stream. Anexemplary transition can comprise, for example, fading from one sourceto another. An effect object can operate on a single source or on acomposite of sources. An effect essentially receives a single inputstream, operates on it in some way, and produces a single output stream.An exemplary effect can comprise a black-and-white effect in which avideo stream that is configured for presentation in color format isrendered into a video stream that is configured for presentation inblack and white format. Unlike conventional effect filters, effectobject 906 may well perform multiple effect tasks. That is, inaccordance with one implementation, an effect object (e.g., 906) mayactually perform multiple tasks on the received input stream, whereinsaid tasks would require multiple effect filters in a conventionalfilter graph system.

An exemplary user interface 908 is shown and represents what a usermight see when they produce a multimedia project with software executingon a computer. In this example, the user has selected three sources A,B, and C, and has assembled the sources into a project timeline. Theproject timeline defines when the individual sources are to be rendered,as well as when any transitions and/or effects are to occur.

In the discussion that follows, the notion of a track is introduced. Atrack can contain one or more sources or source clips. If a trackcontains more than one source clip, the source clips cannot overlap. Ifsource clips are to overlap (e.g. fading from one source to another, orhaving one source obscure another), then multiple tracks are used. Atrack can thus logically represent a layer on which sequential video isproduced. User interface 908 illustrates a project that utilizes threetracks, each of which contains a different source. In this particularproject source A will show for a period of time. At a defined time inthe presentation, source A is obscured by source B. At some later time,source B transitions to source C.

In accordance with the described embodiment, the user-defined editingproject 900 is translated into a data structure 910 that represents theproject. In the illustrated and described example, this data structure910 comprises a tree structure. It is to be understood, however, thatother data structures could be used. The use of tree structures torepresent editing projects is well-known and is not described here inany additional detail. Once the data structure 910 is defined, it isprocessed to provide a data structure 912 that is utilized to programthe matrix switch. In the illustrated and described embodiment, datastructure 912 comprises a grid from which the matrix switch can beprogrammed. It is to be understood and appreciated that other datastructures and techniques could, however, be used to program the matrixswitch without departing from the spirit and scope of the claimedsubject matter.

The processing that takes place to define data structures 910 and 912can take place using any suitable hardware, software, firmware, orcombination thereof. In the examples set forth below, the processingtakes place utilizing software in the form of a video editing softwarepackage that is executable on a general purpose computer.

Example Project

For purposes of explanation, consider FIG. 10 which shows project 908from FIG. 9 in a little additional detail. Here, a time line containingnumbers 0-16 is provided adjacent the project to indicate whenparticular sources are to be seen and when transitions and effects (whenpresent) are to occur. In the examples in this document, the followingconvention exists with respect to projects, such as project 908. Apriority exists for video portions of the project such that as oneproceeds from top to bottom, the priority increases. Thus, in the FIG.10 example, source A has the lowest priority followed by source B andsource C. Thus, if there is an overlap between higher and lower prioritysources, the higher priority source will prevail. For example, source Bwill obscure source A from between t=4-8.

In this example, the following can be ascertained from the project 908and time line: from time t=0-4 source A should be routed to the matrixswitch's primary output; from t=4-12 source B should be routed to thematrix switch's primary output; from t=12-14 there should be atransition between source B and source C which should be routed to thematrix switch's primary output; and from t=14-16 source C should berouted to the matrix switch's primary output. Thus, relative to thematrix switch, each of the sources and the transition can becharacterized by where it is to be routed at any given time. Consider,for example, the table just below: Object Routing for a given time C t =0-12 (nowhere); t = 12-14 (transition); t = 14-16 (primary output) B t =0-4 (nowhere); t = 4-12 (primary output); t = 12-14 (transition); t =14-16 (nowhere) A t = 0-4 (primary output); t = 4-16 (nowhere)Transition t = 0-12 (nowhere); t = 12-14 (primary output); t = 14-16(nowhere)

FIG. 11 shows an exemplary matrix switch 1100 that can be utilized inthe presentation of the user's project. Matrix switch 1100 comprisesmultiple inputs and multiple outputs. Recall that a characteristic ofthe matrix switch 1100 is that any of the inputs can be routed to any ofthe outputs at any given time. A transition element 1102 is provided andrepresents the transition that is to occur between sources B and C.Notice that the matrix switch includes four inputs numbered 0-3 andthree outputs numbered 0-2. Inputs 0-2 correspond respectively tosources A-C, while input 3 corresponds to the output of the transitionelement 1102. Output 0 corresponds to the switch's primary output, whileoutputs 1 and 2 are routed to the transition element 1102.

The information that is contained in the table above is the informationthat is utilized to program the matrix switch. The discussion presentedbelow describes but one implementation in which the informationcontained in the above table can be derived from the user's project timeline.

Recall that as a user edits or creates a project, software thatcomprises a part of their editing software builds a data structure thatrepresents the project. In the FIG. 9 overview, this was data structure910. In addition to building the data structure that represents theediting project, the software also builds and configures a matrix switchthat is to be used to define the output stream that embodies theproject. Building and configuring the matrix switch can include buildingthe appropriate graphs (e.g., a collection of software objects, orfilters) that are associated with each of the sources and associatingthose graphs with the correct inputs of the matrix switch. In addition,building and configuring the matrix switch can also include obtainingand incorporating additional appropriate filters with the matrix switch,e.g. filters for transitions, effects, and mixing (for audio streams).This will become more apparent below.

FIG. 12 shows a graphic representation of an exemplary data structure1200 that represents the project of FIG. 10. Here, the data structurecomprises a traditional hierarchical tree structure. Any suitable datastructure can, however, be utilized. The top node 1202 constitutes agroup node. A group encapsulates a type of media. For example, in thepresent example the media type comprises video. Another media type isaudio. The group node can have child nodes that are either tracks orcomposites. In the present example, three track nodes 1204, 1206, and1208 are shown. Recall that each track can have one or more sources. Ifa track comprises more than one source, the sources cannot overlap.Here, all of the sources (A, B, and C) overlap. Hence, three differenttracks are utilized for the sources. In terms of priority, the lowestpriority source is placed into the tree furthest from the left at 1204a. The other sources are similarly placed. Notice that source C (1208 a)has a transition 1210 associated with it. A transition object, in thisexample, defines a two-input/one output operation. When applied to atrack or a composition (discussed below in more detail), the transitionobject will operate between the track to which it has been applied, andany objects that are beneath it in priority and at the same level in thetree. A “tree level” has a common depth within the tree and belongs tothe same parent. Accordingly, in this example, the transition 1210 willoperate on a source to the left of the track on which source C resides,and beneath it in priority, i.e. source B. If the transition is appliedto any object that has nothing beneath it in the tree, it willtransition from blackness (and/or silence if audio is included).

Once a data structure representing the project has been built, in thiscase a hierarchical tree structure, a rendering engine processes thedata structure to provide another data structure that is utilized toprogram the matrix switch. In the FIG. 9 example, this additional datastructure is represented at 912. It will be appreciated and understoodthat the nodes of tree 1200 can include so-called meta information suchas a name, ID, and a time value that represents when that particularnode's object desires to be routed to the output, e.g. node 1204 a wouldinclude an identifier for the node associating it with source A, as wellas a time value that indicates that source A desires to be routed to theoutput from time t=0-8. This meta information is utilized to build thedata structure that is, in turn, utilized to program the matrix switch.

In the example about to be described below, a specific data structure inthe form of a grid is utilized. In addition, certain specifics aredescribed with respect to how the grid is processed so that the matrixswitch can be programmed. It is to be understood that the specificdescribed approach is for exemplary purposes only and is not intended tolimit application of the claims. Rather, the specific approachconstitutes but one way of implementing broader conceptual notionsembodied by the inventive subject matter.

FIGS. 13-18 represent a process through which the inventive grid isbuilt. In the grid about to be described, the x axis represents time,and the y axis represents layers in terms of priority that go fromlowest (at the top of the grid) to highest (at the bottom of the grid).Every row in the grid represents the video layer. Additionally, entriesmade within the grid represent output pins of the matrix switch. Thiswill become apparent below.

The way that the grid is built in this example is that the renderingengine does a traversal operation on the tree 1200. In this particularexample, the traversal operation is known as a “depth-first,left-to-right” traversal. This operation will layerize the nodes so thatthe leftmost track or source has the lowest priority and so on. Doingthe above-mentioned traversal on tree 1200 (FIG. 12), the first nodeencountered is node 1204 which is associated with source A. This is thelowest priority track or source. A first row is defined for the grid andis associated with source A. After the first grid row is defined, a gridentry is made and represents the time period for which source A desiresto be routed to the matrix switch's primary output.

FIG. 13 shows the state of a grid 1300 after this first processing step.Notice that from time t=0-8, a “0” has been placed in the grid. The “0”represents the output pin of the matrix switch—in this case the primaryoutput. Next, the traversal encounters node 1206 (FIG. 12) which isassociated with source B. A second row is thus defined for the grid andis associated with source B. After the second grid row is defined, agrid entry is made and represents the time period for which source Bdesires to be routed to the matrix switch's primary output.

FIG. 14 shows the state of grid 1300 after this second processing step.Notice that from time t=4-14, a “0” has been placed in the grid. Noticeat this point that something interesting has occurred which will beresolved below. Each of the layers has a common period of time (i.e.t=4-8) for which it desires to be routed to the matrix switch's primaryoutput. However, because of the nature of the matrix switch, only oneinput can be routed to the primary output at a time. Next, the traversalencounters node 1208 (FIG. 12) which is associated with source C. Inthis particular processing example, a rule is defined that sources ontracks are processed before transitions on the tracks are processedbecause transitions operate on two objects that are beneath them. Athird row is thus defined for the grid and is associated with source C.After the third row is defined, a grid entry is made and represents thetime period for which source C desires to be routed to the matrixswitch's primary output.

FIG. 15 shows the state of grid 1300 after this third processing step.Notice that from time t=12-16, a “0” has been placed in the grid. Next,the traversal encounters node 1210 (FIG. 12) which corresponds to thetransition. Thus, a fourth row is defined in the grid and is associatedwith the transition. After the fourth row is defined, a grid entry ismade and represents the time period for which the transition desires tobe routed to the matrix switch's primary output.

FIG. 16 shows the state of grid 1300 after this fourth processing step.Notice that from time t=12-14, a “0” has been placed in the grid for thetransition entry. The transition is a special grid entry. Recall thatthe transition is programmed to operate on two inputs and provide asingle output. Accordingly, starting at the transition entry in the gridand working backward, each of the entries corresponding to the same treelevel are examined to ascertain whether they contain entries thatindicate that they want to be routed to the output during the same timethat the transition is to be routed to the output. If grid entries arefound that conflict with the transition's grid entry, the conflictinggrid entry is changed to a value to corresponds to an output pin thatserves as an input to the transition element 1102 (FIG. 11). This isessentially a redirection operation. In the illustrated grid example,the transition first finds the level that corresponds to source C. Thislevel conflicts with the transition's grid entry for the time periodt=12-14. Thus, for this time period, the grid entry for level C ischanged to a switch output that corresponds to an input for thetransition element. In this example, a “2” is placed in the grid tosignify that for this given time period, this input is routed to outputpin 2. Similarly, continuing up the grid, the next level that conflictswith the transition's grid entry is the level that corresponds to sourceB. Thus, for the conflicting time period, the grid entry for level B ischanged to a switch output that corresponds to an input for thetransition element. In this example, a “1” is placed in the grid tosignify that for this given time period, this input is routed to outputpin 1 of the matrix switch.

FIG. 17 shows the state of the grid at this point in the processing.Next, a pruning function is implemented which removes any other lowerpriority entry that is contending for the output with a higher priorityentry. In the example, the portion of A from t=4-8 gets removed becausethe higher priority B wants the output for that time.

FIG. 18 shows the grid with a cross-hatched area that signifies thatportion of A's grid entry that has been removed.

At this point, the grid is in a state in which it can be used to programthe matrix switch. The left side entries—A, B, C, and TRANS representinput pin numbers 0, 1, 2, and 3 (as shown) respectively, on the matrixswitch shown in FIG. 11. The output pin numbers of the matrix switch aredesignated at 0, 1, and 2 both on the switch in FIG. 11 and within thegrid in FIG. 18. As one proceeds through the grid, starting with sourceA, the programming of the matrix switch can be ascertained as follows: Ais routed to output pin 0 of the matrix switch (the primary output) fromt=0-4. From t=4-16, A is not routed to any output pins. From t=0-4, B isnot routed to any of the output pins of the matrix switch. From t=4-12,B is routed to the primary output pin 0 of the matrix switch. Fromt=12-14, B is routed to output pin 1 of the matrix switch. Output pin 1of the matrix switch corresponds to one of the input pins for thetransition element 1102 (FIG. 11). From t=14-16, B is not routed to anyof the output pins of the matrix switch. From t=0-12, C is not routed toany of the output pins of the matrix switch. From t=12-14, C is routedto output pin 2 of the matrix switch. Output pin 2 of the matrix switchcorresponds to one of the input pins for the transition element 302(FIG. 3). From t=12-14 the transition element (input pin 3) is routed tooutput pin 0. From t=14-16, C is routed to output pin 0 of the matrixswitch.

As alluded to above, one of the innovative aspects of the matrix switch308 is its ability to seek to any point in a source, without having toprocess the intervening content serially through the filter. Rather,matrix switch 308 identifies an appropriate transition point and dumpsat least a subset of the intervening content, and continues processingfrom the seeked point in the content.

The ability of the matrix switch 308 to seek to any point in the mediacontent gives rise to certain performance enhancement heretoforeunavailable in computer implemented media processing systems. Forexample, generation of a filter graph by render engine 222 may take intoaccount certain performance characteristics of the media processingsystem which will execute the user-defined media processing project. Inaccordance with this example implementation, render engine 222 mayaccess and analyze the system registry of the operating system, forexample, to ascertain the performance characteristics of hardware and/orsoftware elements of the computing system implementing the mediaprocessing system, and adjust the filter graph construction to improvethe perceived performance of the media processing system by the user.Nonetheless, there will always be a chance that a particular instance ofa filter graph will not be able to process the media stream fast enoughto provide the desired output at the desired time, i.e., processing ofthe media stream bogs down leading to delays at the rendering filter. Insuch a case, matrix switch 308 will recognize that it is not receivingmedia content at the appropriate project time, and may skip certainsections of the project in an effort to “catch-up” and continue theremainder of the project in real time. According to one implementation,when matrix switch 308 detects such a lag in processing, it will analyzethe degree of the lag and issue a seek command to the source (throughthe source processing chain) to a future point in the project, whereprocessing continues without processing any further content prior to theseeked point.

Thus, for the editing project depicted in FIG. 10, the processingdescribed above first builds a data structure (i.e. data structure 1200in FIG. 12) that represents the project in hierarchical space, and thenuses this data structure to define or create another data structure thatcan be utilized to program the matrix switch.

FIG. 19 is a flow diagram that describes steps in a method in accordancewith the described embodiment. The method can be implemented in anysuitable hardware, software, firmware, or combination thereof. In theillustrated and described embodiment, the method is implemented insoftware.

Step 1900 provides a matrix switch. An exemplary matrix switch isdescribed above. Step 1902 defines a first data structure thatrepresents the editing project. Any suitable data structure can be used,as will be apparent to those of skill in the art. In the illustrated anddescribed embodiment, the data structure comprises a hierarchical treestructure having nodes that can represent tracks (having one or moresources), composites, transitions and effects. Step 1904 processes thefirst data structure to provide a second data structure that isconfigured to program the matrix switch. Any suitable data structure canbe utilized to implement the second data structure. In the illustratedand described embodiment, a grid structure is utilized. Exemplaryprocessing techniques for processing the first data structure to providethe second data structure are described above. Step 1906 then uses thesecond data structure to program the matrix switch.

Example Project with a Transition and an Effect

Consider project 2000 depicted in FIG. 20. In this project there arethree tracks, each of which contains a source, i.e. source A, B and C.This project includes an effect applied on source B and a transitionbetween sources B and C. The times are indicated as shown.

As the user creates their project, a data structure representing theproject is built. FIG. 21 shows an exemplary data structure in the formof a hierarchical tree 2100 that represents project 2000. There, thedata structure includes three tracks, each of which contains one of thesources. The sources are arranged in the tree structure in the order oftheir priority, starting with the lowest priority source on the left andproceeding to the right. There is an effect (i.e. “Fx”) that is attachedto or otherwise associated with source B. Additionally, there is atransition attached to or otherwise associated with source C.

In building the grid for project 2000, the following rule is employedfor effects. An effect, in this example, is a one-input/one-outputobject that is applied to one object—in this case source B. When theeffect is inserted into the grid, it looks for any one object beneath itin priority that has a desire to be routed to the primary output of thematrix switch at the same time. When it finds a suitable object, itredirects that object's output from the matrix switch's primary outputto an output associated with the effect.

As an example, consider FIG. 22 and the grid 2200. At this point in theprocessing of tree 2100, the rendering engine has incorporated entriesin the grid corresponding to sources A, B and the effect. It has done soby traversing the tree 2100 in the above-described way. In this example,the effect has already looked for an object beneath it in priority thatis competing for the primary output of the matrix switch. It found anentry for source B and then redirected B's grid entry to a matrix switchoutput pin that corresponds to the effect—here output pin 1.

As the render engine 222 completes its traversal of tree 2100, itcompletes the grid. FIG. 23 shows a completed grid 2200. Processing ofthe grid after that which is indicated in FIG. 22 takes placesubstantially as described above with respect to the first example.Summarizing, this processing though: after the effect is entered intothe grid and processed as described above, the traversal of tree 2100next encounters the node associated with source C. Thus, a row is addedin the grid for source C and an entry is made to indicate that source Cdesires the output from t=12-16. Next, the tree traversal encounters thenode associated with the transition. Accordingly, a row is added to thegrid for the transition and a grid entry is made to indicate that thetransition desires the output from t=12-14. Now, as described above, thegrid is examined to find two entries, lower in priority than thetransition and located at the same tree level as the transition, thatcompete for the primary output of the matrix switch. Here, those entriescorrespond to the grid entries for the effect and source C that occurfrom t=12-14. These grid entries are thus redirected to output pins ofthe matrix switch 308 that correspond to the transition-here pins 2 and3 as indicated. Next, the grid is pruned which, in this example, removesa portion of the grid entry corresponding to source A for t=4-8 becauseof a conflict with the higher-priority entry for source B.

FIG. 24 shows the resultant matrix switch that has been built andconfigured as the grid was being processed above. At this point, thegrid can be used to program the matrix switch. From the grid picture, itis very easy to see how the matrix switch 308 is going to be programmed.Source A will be routed to the matrix switch's primary output (pin 0)from t=0-4; source B will be redirected to output pin 1 (effect) fromt=4-14 and the effect on B will be routed to the output pin 0 fromt=4-12. From t=12-14, the effect and source C will be routed to outputpins corresponding to the transition (pins 2 and 3) and, accordingly,during this time the transition (input pin 4) will be routed to theprimary output (output pin 0) of the matrix switch. From t=14-16, sourceC will be routed to the primary output of the matrix switch.

It will be appreciated that as the software, in this case the renderengine 222, traverses the tree structure that represents a project, italso builds the appropriate graphs and adds the appropriate filters andgraphs to the matrix switch. Thus, for example, as the render engine 222encounters a tree node associated with source A, in addition to addingan entry to the appropriate grid, the software builds the appropriategraphs (i.e. collection of linked filters), and associates those filterswith an input of the matrix switch. Similarly, when the render engine222 encounters an effect node in the tree, the software obtains aneffect object or filter and associates it with the appropriate output ofthe matrix switch. Thus, in the above examples, traversal of the treestructure representing the project also enables the software toconstruct the appropriate graphs and obtain the appropriate objects andassociate those items with the appropriate inputs/outputs of the matrixswitch 308. Upon completion of the tree traversal and processing of thegrid, an appropriate matrix switch has been constructed, and theprogramming (i.e. timing) of inputs to outputs for the matrix switch hasbeen completed.

Treatment of “blanks” in a Project

There may be instances in a project when a user leaves a blank in theproject time line. During this blank period, no video or audio isscheduled for play.

FIG. 25 shows a project that has such a blank incorporated therein. Ifthere is such a blank left in a project, the software is configured toobtain a “black” source and associate the source with the matrix switchat the appropriate input pin. The grid is then configured when it isbuilt to route the black source to the output at the appropriate timesand fade from the black (and silent) source to the next source at theappropriate times. The black source can also be used if there is atransition placed on a source for which there is no additional sourcefrom which to transition.

Audio Mixing

In the examples discussed above, sources comprising video streams werediscussed. In those examples, at any one time, only two video streamswere combined into one video stream. However, each project can, andusually does contain an audio component. Alternately, a project cancontain only an audio component. The audio component can typicallycomprise a number of different audio streams that are combined. Thediscussion below sets forth but one way of processing and combiningaudio streams.

In the illustrated example, there is no limit on the number of audiostreams that can be combined at any one time.

Suppose, for example, there is an audio project that comprises 5 tracks,A-E. FIG. 26 shows an exemplary project. The shaded portions of eachtrack represent the time during which the track is not playing. So, forexample, at t=0-4, tracks B, D, and E are mixed together and will play.From t=4-10, tracks A-E are mixed together and will play, and the like.

FIG. 27 shows the grid for this project at 2700. Since we are dealingwith this composition now, all of the effects and transitions includingthe audio mixing are only allowed to affect things in this composition.Thus, there is the concept of a boundary 2702 that prevents any actionsor operations in this composition from affecting any other grid entries.Note that there are other entries in the grid and that thepresently-illustrated entries represent only those portions of theproject that relate to the audio mixing function.

Grid 2700 is essentially set up in a manner similar to that describedabove with respect to the video projects. That is, for each track, a rowis added to the grid and a grid entry is made for the time period duringwhich the source on that track desires to be routed to the primaryoutput of the matrix switch. In the present example, grid entries aremade for sources A-E. Next, in the same way that a transition or effectwas allocated a row in the grid, a “mix” element is allocated a row inthe grid as shown and a grid entry is made to indicate that the mixelement desires to be routed to the primary output of the matrix switchfor a period of time during which two or more sources compete for thematrix switch's primary output. Note that in this embodiment, allocationof a grid row for the mix element can be implied. Specifically, whereasin the case of a video project, overlapping sources simply result inplaying the higher priority source (unless the user defines a transitionbetween them), in the audio realm, overlapping sources are treated as animplicit request to mix them. Thus, the mix element is allocated a gridrow any time there are two or more overlapping sources.

Once the mix element is allocated into the grid, the grid is processedto redirect any conflicting source entries to matrix switch output pinsthat correspond to the mix element. In the above case, redirection ofthe grid entries starts with pin 3 and proceeds through to pin 7. Thecorresponding matrix switch is shown in FIG. 28. Notice that all of thesources are now redirected through the mix element which is amulti-input/one output element. The mix element's output is fed backaround and becomes input pin 15 of the matrix switch. All of theprogramming of the matrix switch is now reflected in the grid 2700.Specifically, for the indicated time period in the grid, each of thesources is routed to the mix element which, in turn, mixes theappropriate audio streams and presents them to the primary output pin 0of the matrix switch.

Compositions

There are situations that can arise when building an editing projectwhere it would be desirable to apply an effect or a transition on just asubset of a particular project or track. Yet, there is no practicableway to incorporate the desired effect or transition. In the past,attempts to provide added flexibility for editing projects have beenmade in the form of so called “bounce tracks”, as will be appreciatedand understood by those of skill in the art. The use of bounce tracksessentially involves processing various video layers (i.e. tracks),writing or moving the processed layers or tracks to another location,and retrieving the processed layers when later needed for additionalprocessing with other layers or tracks. This type of processing can beslow and inefficient.

To provide added flexibility and efficiency for multi-media editingprojects, the notion of a composite or composition is introduced. Acomposite or composition can be considered as a representation of anediting project as a single track. Recall that editing projects can haveone or more tracks, and each track can be associated with one or moresources that can have effects applied on them or transitions betweenthem. In addition, compositions can be nested inside one another.

Example Project with Composite

Consider, for example, FIG. 29 which illustrates an exemplary project2900 having a composition 2902. In this example, composition 2902comprises sources B and C and a transition between B and C that occursbetween t=12-14. This composition is treated as an individual track orlayer. Project 2900 also includes a source A, and a transition betweensource A and composition 2902 at t=4-8. It will be appreciated thatcompositions can be much more complicated than the illustratedcomposition, which is provided for exemplary purposes only. Compositionsare useful because they allow the grouping of a particular set ofoperations on one or more tracks. The operation set is performed on thegrouping, and does not affect tracks that are not within the grouping.To draw an analogy, a composition is similar in principle to amathematical parenthesis. Those operations that appear within theparenthesis are carried out in conjunction with those operations thatare intended to operate of the subject matter of the parenthesis. Theoperations within the parenthesis do not affect tracks that do notappear within the parenthesis.

In accordance with the processing that is described above in connectionwith FIG. 19, a first data structure is defined that represents theediting project. FIG. 30 shows an exemplary data structure 3000 in theform of a hierarchical tree structure. In this example, group node 3002includes two children—track node 3004 and composite node 3006. Tracknode 3004 is associated with source A. Composite node 3006 includes twochildren—track nodes 3008 and 3010 that are respectively associated withsources B (3008 a) and C (3010 a). A transition T2 (3012) is applied onsource C and a transition T1 (3014) is applied on composition 3006.

Next, data structure 3000 is processed to provide a second datastructure that is configured to program the matrix switch. Note that asthe data structure is being programmed, a matrix switch is being builtand configured at the same time. In this example, the second datastructure comprises a grid structure that is assembled in much the sameway as was described above. There are, however, some differences and,for purposes of understanding, the complete evolution of the gridstructure is described here. In the discussion that follows, thecompleted matrix switch is shown in FIG. 38.

When the rendering engine initiates the depth-first, left-to-righttraversal of data structure 3000, the first node it encounters is tracknode 3004 which is associated with source A. Thus, a first row of thegrid is defined and a grid entry is made that represents the time periodfor which source A desires to be routed to the matrix switch's primaryoutput pin.

FIG. 31 shows the state of a grid 3100 after this first processing step.Next the traversal of data structure 3000 encounters the composite node3006. The composite node is associated with two tracks—track 3008 andtrack 3010. Track 3008 is associated with source B. Accordingly, asecond row of the grid is defined and a grid entry is made thatrepresents the time period for which source B desires to be routed tothe matrix switch's primary output pin. Additionally, since B is amember of a composition, meta-information is contained in the grid thatindicates that this grid row defines one boundary of the composition.This meta-information is graphically depicted with a bracket thatappears to the left of the grid row.

FIG. 32 shows the state of grid 3100 after this processing step. Next,the traversal of data structure 3000 encounters node 3010 which isassociated with source C. Thus, a third row of the grid is added and agrid entry is made that represents the time period for which source Cdesires to be routed to the matrix switch's primary output pin.

FIG. 33 shows the state of grid 3100 after this processing step. Noticethat the bracket designating the composition now encompasses the gridrow associated with source C. The traversal next encounters node 3012which is the node associated with the second transition T2. Thus, as inthe above example, a grid row is added for the transition and a gridentry is made that represents the time period for which the transitiondesires to be routed to the matrix switch's primary output pin.

FIG. 34 shows the state of grid 3100 after this processing step. Noticethat the bracket designating the composition is now completed andencompasses grid row entries that correspond to sources B and C and thetransition between them. Recall from the examples above that atransition, in this example, is programmed to operate on two inputs andprovide a single output. In this instance, and because the transitionoccurs within a composition, the transition is constrained by a rulethat does not allow it to operate on any elements outside of thecomposition. Thus, starting at the transition entry and working backwardthrough the grid, entries at the same tree level and within thecomposition (as designated by the bracket) are examined to ascertainwhether they contain entries that indicate that they want to be routedto the output during the same time that the transition is to be routedto the output. Here, both of the entries for sources B and C haveportions that conflict with the transition's entry. Accordingly, thoseportions of the grid entries for sources B and C are redirected orchanged to correspond to output pins that are associated with atransition element that corresponds to transition T2.

FIG. 35 shows the state of grid 3100 after this processing step. Thetraversal next encounters node 3014 which is the node that is associatedwith the transition that occurs between source A and composition 2902(FIG. 29). Processing of this transition is similar to processing of thetransition immediately above except for the fact that the transitiondoes not occur within the composition. Because the transition occursbetween the composition and another source, one of the inputs for thetransition will be the composition, and one of the inputs will be sourceA (which is outside of the composition). Thus, a grid row is added forthis transition and a grid entry is made that represents the time periodfor which the transition desires to be routed to the matrix switch'sprimary output pin.

FIG. 36 shows the state of grid 3100 after this processing step. At thispoint then, the grid is examined for entries that conflict with theentry for transition T1. One conflicting grid entry is found for the rowthat corresponds to source B (inside the composition) and one thatcorresponds to source A (outside the composition). Accordingly, thoseportions of the grid row that conflict with transition T1 are changed orredirected to have values that are associated with output pins of thematrix switch that are themselves associated with a transition elementT1. In this example, redirection causes an entry of “3” and “4” to beinserted as shown.

FIG. 37 shows the state of grid 3100 after this processing step. Ifnecessary, a pruning operation would further ensure that the grid has nocompeting entries for the primary output of the matrix switch. Theassociated input pin numbers of the matrix switch are shown to the leftof grid 3100.

FIG. 38 shows a suitably configured matrix switch that has been build inaccordance with the processing described above. Recall that, as datastructure 3000 (FIG. 30) is processed by the rendering engine, a matrixswitch is built and configured in parallel with the building andprocessing of the grid structure that is utilized to program the matrixswitch. From the matrix switch and grid 3100 of FIG. 37, the programmingof the switch can be easily ascertained.

FIG. 38 a shows an exemplary data structure that represents a projectthat illustrates the usefulness of composites. In this example, theproject can mathematically be represented as follows:(Fx−noisy(ATx−Blend B))Tx−Blend C

Here, an effect (noisy) is applied to A blended with B, the result ofwhich is applied to a blend with C. The composite in this example allowsthe grouping of the things beneath it so that the effect (noisy), whenit is applied, is applied to everything that is beneath it. Notice thatwithout the composite node, there is no node where an effect can beapplied that will affect (A Tx−Blend B). Hence, in this example,operations that appear within the parenthesis are carried out on tracksthat appear within the parenthesis. Those operations do not affecttracks that are not within the parenthesis.

FIG. 39 is a flow diagram that described steps in a method in accordancewith one embodiment. The method can be implemented in any suitablehardware, software, firmware, or combination thereof. In thepresently-described example, the method is implemented in software.

Step 3900 defines a multimedia editing project that includes at leastone composite. The composite represents multiple tracks as a singletrack for purposes of the processing described just below. It isimportant to note that, in the processing described just below, andbecause of the use of composites, the extra processing that is requiredby bounce tracks is avoided (i.e. operating on two tracks, moving theoperation result to another location, and retrieving the operationresult when later needed). This reduces the processing time that isrequired to render a multi-media project. Step 3902 defines a first datastructure that represents the editing project. Any suitable datastructure can be utilized. In the present example, a data structure inthe form of a hierarchical tree is utilized. An exemplary tree is shownin FIG. 30. Step 3904 processes the first data structure to provide asecond data structure that is configured to program a matrix switch. Inthe illustrated example, the second data structure comprises a gridstructure. Exemplary processing is described in the context of FIGS.30-37. Step 3906 then programs the matrix switch using the second datastructure.

Multi-Media File Locator Object

In some instances, the environment in which multi-media editing projectsare created is a file sharing environment in which the multi-media filesthat can be used for particular projects are shared among multipleusers. Typically, in these environments, the files are maintained in alarge network-accessible database or storage facility. When a userwishes to use a particular multi-media file in an editing project, theywill typically retrieve the file from the network and incorporate thefile into their project. When many users are part of this sharingenvironment, significant slowdowns can be experienced when the usersattempt to run their projects off of the network.

As an aside, multi-media files are typically very large files, e.g. 100Megabytes to 8 Gigabytes. Additionally, multi-media files themselves arenot generally changed by the users. Rather, the users use the files intheir multi-media editing projects which themselves can change from userto user. Thus, in this environment, the types of files that are employedare typically large, unchanging files. Accordingly, it is easy tounderstand and appreciate, from the size of these files, the networkslowdowns that can be caused when multiple users attempt to run theirprojects off of the network.

FIG. 40 shows but one exemplary system 4000 in which the inventivetechniques described below can be implemented. System 4000 includes anetwork 4002 which can be any suitable network, e.g. LAN, WAN and thelike. A multi-media file storage facility 4004 is provided and isaccessible via the network 4002. A number of different user computersare provided, with exemplary computers 4006, 4008, and 4010 being shown.Each user computer has an associated local storage mechanism, e.g. ahard drive.

Each of the user computers typically executes a multi-media editingapplication which allows a user to build a multi-media editing projectas described above. The inventive techniques described below permit auser to retrieve one or more multi-media files from a network accessiblestorage location and maintain the files locally, e.g. in directories ontheir hard drive. When they then request particular multi-media filesfor use, the editing application can first check one or more localdirectories for the requested files, rather than checking the network.If the requested files are not found locally, then the editingapplication can check the network-accessible file locations for therequested file. By maintaining the multi-media files locally, networkslowdown issues can be mitigated.

FIG. 41 is a flow diagram that describes steps in a method in accordancewith the described embodiment. The method can be implemented in anysuitable hardware, software, firmware or combination thereof. In thedescribed embodiment, the method is implemented in software. In oneparticular embodiment that is described in more detail below, the methodis implemented in connection with a multi-media file locator objectcomprising a COM object.

Step 4100 generates a request for a network-maintained multi-media file.This request is ordinarily generated by a multi-media editingapplication executing on a user computer, such as any one of computers4006-4010 (FIG. 40). Step 4102 intercepts the request locally. Step 4104then ascertains whether the file or files referenced in the requestexist. Step 4106 determines one or more local directories wheremulti-media files are, or have been maintained. Specifically, as a userretrieves and uses multi-media files, they can store them locally. Whenthey store them locally, they typically have certain designateddirectories that contain the files, e.g. “C:/myfiles/multimedia_files”.Alternately, they might store the files anywhere on their hard drive. Auser can then designate appropriate directories as directories thatcontain multi-media files. One example of when a user can do this isgiven below. Thus, when a request is intercepted, as in step 4102, thesoftware can quickly ascertain the directories of interest (e.g. thedirectories that have been designated by a user as containingmulti-media files) that are likely to contain the multi-media files.Step 4108 then checks the determined local directories for the requestedmulti-media file. This step is advantageous in that it can avoidchecking all of the directories of a hard drive which can be timeconsuming. If the file is found locally (step 4110), then the file isretrieved from its local location and used (step 4112). If, on the otherhand, the file is not found locally, step 4114 asks the user to point toa local directory where the multi-media file might be stored. Step 4116then checks the user-designated directory and if the multi-media file isfound, step 4112 retrieves and uses the file. If, on the other hand,step 4116 does not find the file in the local directory designated bythe user, step 4118 checks appropriate network directories for therequested multi-media file. This step can be implemented by sending onthe request that was intercepted at step 4102. It will be appreciatedand understood that once a user designates a new local directory (i.e.responsive to step 4114) that contains one multi-media file, thesoftware will remember this directory and will automatically check itwhen multi-media files are requested in the future.

Although the above method can be implemented in any suitable hardware,software, firmware, or combination thereof, in one implementation themethod is implemented with file locator object using object orientedprogramming techniques. Although the file locator object can compriseany suitable object type, the described object is a COM object.

FIG. 42 shows an exemplary multi-media file locator object 4200. Thelocator object 4200 is programmed to intercept network-bound requests4202 for multi-media files, and determine whether the requested file orfiles reside locally in a local directory 4204. To do this, the locatorobject maintains a directory list 4206 that it uses to keep track of thelocal directories that are currently used to store multi-media files, orthat have been used in the past to store such files. Once the locatorobject ascertains which directories need to be checked, it simply checksthe listed directories for the file(s) of interest. It will beappreciated that the directory list 4206 can be updated whenever theuser retrieves and stores a new multi-media file locally. One way ofdoing this is to ask the user to designate one or more directories inwhich the multi-media files are stored. The directory list can also beupdated when a user is asked by the software to point to a directorywhere a requested-but-not-found multi-media file has been stored. If,upon checking the local directories (both those on the list and thosesubsequently designated by the user), the locator object 4200 fails tolocate the file of interest, it can then route the intercepted filerequest to the network so that the network-accessible multi-media filescan be searched. If the requested file is found on the network, when thefile is retrieved, an addition can be made to the directory list 4206 toindicate the local location of the file, if the local location isdifferent from any of the locations already maintained on the list. Inthe described embodiment, the user can add directories to the list bysimply designating additional directories where they have storedmulti-media files.

CONCLUSION

The described embodiments can be used to provide improvements overprevious multi-media editing systems. Various efficiencies are achievedthat reduce the processing times and can thereby improve the userexperience when using multi-media project editing software applications.

Although the invention has been described in language specific tostructural features and/or methodological steps, it is to be understoodthat the invention defined in the appended claims is not necessarilylimited to the specific features or steps described. Rather, thespecific features and steps are disclosed as preferred forms ofimplementing the claimed invention.

1. A method of processing a multi-media editing project comprising:providing a software-implemented matrix switch that is programmable toroute multiple switch inputs to multiple switch outputs and to produce asingle output that provides a data stream defined by a multi-mediaediting project; generating a request for one or more multi-media filesfor use in the multi-media editing project, the request being generatedby a user computer that comprises part of a network where multi-mediafiles are maintained in a network—accessible location; intercepting therequest; ascertaining whether a requested multi-media file is located onthe user computer; retrieving the multi-media file if the file islocated on the user computer; seeking the requested file from thenetwork-accessible location if the multi-media file is not located onthe user computer; and producing a multi-media editing project using thesoftware-implemented matrix switch.
 2. The method of claim 1 furthercomprising asking a user to designate a local directory if a requestedfile is not found on the user computer.
 3. The method of claim 1 furthercomprising asking a user to designate a local directory if a requestedfile is not found on the user computer, and then searching for therequested file in a designated local directory before seeking therequested file from the network-accessible location.
 4. The method ofclaim 1, wherein said ascertaining comprises checking variouspredetermined file directories on the computer's hard drive.
 5. Themethod of claim 1, wherein said ascertaining comprises: maintaining alist of directories where multi-media files have been stored in thepast; and checking directories on the list for the requested one or morefiles.
 6. The method of claim 1, wherein said ascertaining comprises:maintaining a list of directories where multi-media files are stored;and checking directories on the list for the requested one or morefiles.
 7. The method of claim 1, wherein said ascertaining comprises:maintaining a list of directories where multi-media files have beenstored in the past or are presently stored; and checking directories onthe list for the requested one or more files.
 8. The method of claim 1further comprising: maintaining a list of directories where multi-mediafiles are stored; and updating the list responsive to receiving andstoring a multi-media file in a local directory that is not on the list.9. The method of claim 1 further comprising: maintaining a list ofdirectories where multi-media files are stored; and updating the listresponsive to a user designating a local directory that is not on thelist.
 10. One or more computer-readable media having computer-readableinstructions thereon which, when executed by a computer, implement themethod of claim
 1. 11. A multi-media project editing applicationconfigured for execution on a user computer, the application beingconfigured to implement the method of claim
 1. 12. A method ofprocessing a multi-media editing project comprising: providing asoftware-implemented matrix switch that is programmable to routemultiple switch inputs to multiple switch outputs and to produce asingle output that provides a data stream defined by a multi-mediaediting project; maintaining information on a local computer thatcomprises part of a network having multiple computers, said informationbeing associated with multi-media files that are maintained in anetwork-accessible location and that can be temporarily stored on thelocal computer's hard drive and used to produce a multi-media editingproject; responsive to a request to retrieve a multi-media file from thenetwork-accessible location, using the information to attempt to locatethe requested file on the local computer's hard drive before attemptingto retrieve the file in the network-accessible location; and producing amulti-media editing project using the software-implemented matrixswitch.
 13. The method of claim 12, wherein said information comprises alist of local directories where multi-media files are stored.
 14. Themethod of claim 12, wherein said information comprises a list of localdirectories where multi-media files have been stored.
 15. The method ofclaim 12, wherein said information comprises a list of local directorieswhere multi-media files are or have been stored.
 16. The method of claim12 wherein said information comprises a list of local directories wheremulti-media files are stored, and further comprising asking a user todesignate one or more local directories where a requested multi-mediafile might be stored if the requested file cannot be located in thedirectories designated on the list.
 17. The method of claim 16 furthercomprising updating the list responsive to a user designating the one ormore directories, and searching for multi-media files in the one or moredirectories on subsequent attempts to locate requested files.
 18. One ormore computer-readable media having computer-readable instructionsthereon which, when executed by a computer, implement the method ofclaim
 12. 19. One or more computer-readable media havingcomputer-readable instructions thereon which, when executed by acomputer, cause the computer to: provide a software-implemented matrixswitch that is programmable to route multiple switch inputs to multipleswitch outputs and to produce a single output that provides a datastream defined by a multi-media editing project; maintain a list on alocal computer that comprises part of a network having multiplecomputers, said list being used to determine which local directorieshave been used in the past, or are currently being used to storedmulti-media files that are maintained in a network-accessible location;responsive to a request to retrieve a multi-media file from thenetwork-accessible location, use the list to first attempt to locate therequested file on the local computer's hard drive; and produce amulti-media editing project using the software-implemented matrixswitch.
 20. The computer-readable media of claim 19, wherein theinstructions further cause the computer to attempt to retrieve therequested file in the network-accessible location in the event that thefile cannot be located locally.
 21. The computer-readable media of claim19, wherein the instructions cause the computer to check only thoselocal directories that are contained in the list.
 22. Thecomputer-readable media of claim 19, wherein the instructions cause thecomputer to ask a user to designate a local directory where amulti-media file might be stored in the event a requested file cannot belocated on the local computer's hard drive.
 23. The computer-readablemedia of claim 22, wherein the instructions cause the computer to updatethe list to contain a local directory that the user designates.
 24. Thecomputer-readable media of claim 22, wherein the instructions cause thecomputer to attempt to locate the multi-media file from the designatedlocal directory before attempting to locate the requested multi-mediafile from the network-accessible location.
 25. A method of processing amulti-media editing project comprising: providing a software-implementedmatrix switch that is programmable to route multiple switch inputs tomultiple switch outputs and to produce a single output that provides adata stream defined by a multi-media editing project; receiving one ormore multi-media files from a network-accessible location; locallystoring the one or more multi-media files in a local directory on a usercomputer for use in the multi-media editing project; updating a list oflocal directories that contain or have contained multi-media files inthe past in the event that the one or more multi-media files are storedin a local directory that is not contained in the list; responsive toreceiving a request for a multi-media file that is maintained in thenetwork-accessible location: first checking in all of the localdirectories on the list of local directories for the requested file; andsecond checking the network-accessible location for the requested filein the event the requested file is not found locally; and producing amulti-media editing project using the software-implemented matrixswitch.
 26. The method of claim 25 further comprising prior to secondchecking, asking a user to point to a local directory where therequested file might be stored and checking that local directory for therequested file.
 27. The method of claim 26, wherein said updatingcomprises doing so responsive to a user pointing to a local directorywhere the requested file might be stored.
 28. The method of claim 25,wherein the multi-media files are read only files.
 29. One or morecomputer-readable media having computer-readable instructions thereonwhich, when executed by a computer, implement the method of claim 25.30. One or more computer-readable media having computer-readableinstructions thereon which, when executed by a computer, cause thecomputer to: provide a software-implemented matrix switch that isprogrammable to route multiple switch inputs to multiple switch outputsand to produce a single output that provides a data stream defined by amulti-media editing project; maintain a list of local directories thatare or have been used to store multi-media files on a local usercomputer, the multi-media files being accessible from a network storagelocation; generate a request for a multi-media file that is accessiblefrom a network storage location, the request being intended for use inretrieving a multi-media file from the network accessible storagelocation; intercept the request; ascertain a requested file from therequest; first, determine whether the requested file is locallyavailable by checking all of the local directories maintained on thelist and retrieve the requested file from a local directory if the fileis locally maintained; second, seek the requested file from the networkstorage location if the file is not locally maintained; store therequested file in a local directory if the requested file is retrievedfrom the network storage location; update the list to reflect the localdirectory if the local directory in which the requested file is storedis not on the list; and produce a multi-media editing project using thesoftware-implemented matrix switch.
 31. The computer-readable media ofclaim 30, wherein the instructions cause the computer to update the listresponsive to input received from a user.
 32. A multi-media editingsystem comprising: one or more computer-readable media; matrix switchmeans embodied on the one or more computer-readable media, the matrixswitch means being programmable to route multiple switch inputs tomultiple switch outputs and to produce a single output that provides adata stream defined by a multi-media editing project; a multi-media filelocator object embodied on the one or more computer-readable media andconfigured to intercept network-bound requests for multi-media files anddetermine whether requested files are locally maintained on a usercomputer; and a list embodied on the one or more computer-readable mediaand associated with the file locator object and referencing local filedirectories on the user computer where multi-media files are stored, thelist being used by the file locator object to determine whetherrequested files are locally maintained on the user computer.
 33. Thesystem of claim 32, wherein the list references local file directorieswhere files have been maintained in the past.
 34. The system of claim32, wherein the locator object is configured to update the list.
 35. Thesystem of claim 32, wherein the locator object is configured to updatethe list responsive to a multi-media file being stored in a localdirectory where multi-media files have not been stored before.
 36. Thesystem of claim 32, wherein the locator object is configured to updatethe list responsive to a user designating a local directory where amulti-media file is stored.
 37. The system of claim 32, wherein thelocator object is configured to cause one or more multi-media files tobe sought through the network in the event that the one or more filesare not locally maintained.
 38. The system of claim 32, wherein thelocator object comprises a COM object.
 39. The system of claim 32,wherein the locator object comprises an object-oriented object.